When local isn't enough: extracting distributed rules from networks

Recent work on a network interpretation technique called “banding analysis” has shown how symbolic rules can be easily extracted from trained connectionist networks [2]. This method, however, is limited to “local” interpretations, and cannot take full advantage of the distributed nature of neural networks. In this paper we discuss a technique for extracting distributed symbolic rules from trained connectionist networks. This technique is based on the k-means cluster analysis of activation values across all hidden units instead of single units as in the banding analysis. A novel stopping rule—based on domain specific heuristic information—is defined such that the optimum number of clusters are extracted from the network. Once the appropriate clusters are determined, simple statistics are computed to determine the shared characteristics of all patterns within a cluster. The utility of this technique is illustrated on two benchmark problems. It is concluded that this form of distributed rule extraction is more succinct than simple local analysis of internal structure.